{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LDA模型应用：一眼看穿希拉里的邮件\n",
    "\n",
    "我们拿到希拉里泄露的邮件，跑一把LDA，看看她平时都在聊什么。\n",
    "\n",
    "首先，导入我们需要的一些库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后，把希婆的邮件读取进来。\n",
    "\n",
    "这里我们用pandas。不熟悉pandas的朋友，可以用python标准库csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"HillaryEmails.csv\")\n",
    "# 原邮件数据中有很多Nan的值，直接扔了。\n",
    "df = df[['Id','ExtractedBodyText']].dropna()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 文本预处理：\n",
    "\n",
    "上过我其他NLP课程的同学都知道，文本预处理这个东西，对NLP是很重要的。\n",
    "\n",
    "我们这里，针对邮件内容，写一组正则表达式：\n",
    "\n",
    "（不熟悉正则表达式的同学，直接百度关键词，可以看到一大张Regex规则表）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def clean_email_text(text):\n",
    "    text = text.replace('\\n',\" \") #新行，我们是不需要的\n",
    "    text = re.sub(r\"-\", \" \", text) #把 \"-\" 的两个单词，分开。（比如：pre-processing ==> pre processing）\n",
    "    text = re.sub(r\"\\d+/\\d+/\\d+\", \"\", text) #日期，对主体模型没什么意义\n",
    "    text = re.sub(r\"[0-2]?[0-9]:[0-6][0-9]\", \"\", text) #时间，没意义\n",
    "    text = re.sub(r\"[\\w]+@[\\.\\w]+\", \"\", text) #邮件地址，没意义\n",
    "    text = re.sub(r\"/[a-zA-Z]*[:\\//\\]*[A-Za-z0-9\\-_]+\\.+[A-Za-z0-9\\.\\/%&=\\?\\-_]+/i\", \"\", text) #网址，没意义\n",
    "    pure_text = ''\n",
    "    # 以防还有其他特殊字符（数字）等等，我们直接把他们loop一遍，过滤掉\n",
    "    for letter in text:\n",
    "        # 只留下字母和空格\n",
    "        if letter.isalpha() or letter==' ':\n",
    "            pure_text += letter\n",
    "    # 再把那些去除特殊字符后落单的单词，直接排除。\n",
    "    # 我们就只剩下有意义的单词了。\n",
    "    text = ' '.join(word for word in pure_text.split() if len(word)>1)\n",
    "    return text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "好的，现在我们新建一个colum，并把我们的方法跑一遍："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "docs = df['ExtractedBodyText']\n",
    "docs = docs.apply(lambda s: clean_email_text(s))  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "好，来看看长相："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Thursday March PM Latest How Syria is aiding Qaddafi and more Sid hrc memo syria aiding libya docx hrc memo syria aiding libya docx March For Hillary'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs.head(1).values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们直接把所有的邮件内容拿出来。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "doclist = docs.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LDA模型构建：\n",
    "\n",
    "好，我们用Gensim来做一次模型构建\n",
    "\n",
    "首先，我们得把我们刚刚整出来的一大波文本数据\n",
    "```\n",
    "[[一条邮件字符串]，[另一条邮件字符串], ...]\n",
    "```\n",
    "\n",
    "转化成Gensim认可的语料库形式：\n",
    "\n",
    "```\n",
    "[[一，条，邮件，在，这里],[第，二，条，邮件，在，这里],[今天，天气，肿么，样],...]\n",
    "```\n",
    "\n",
    "引入库："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from gensim import corpora, models, similarities\n",
    "import gensim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了免去讲解安装NLTK等等的麻烦，我这里直接手写一下**停止词列表**：\n",
    "\n",
    "这些词在不同语境中指代意义完全不同，但是在不同主题中的出现概率是几乎一致的。所以要去除，否则对模型的准确性有影响"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "stoplist = ['very', 'ourselves', 'am', 'doesn', 'through', 'me', 'against', 'up', 'just', 'her', 'ours', \n",
    "            'couldn', 'because', 'is', 'isn', 'it', 'only', 'in', 'such', 'too', 'mustn', 'under', 'their', \n",
    "            'if', 'to', 'my', 'himself', 'after', 'why', 'while', 'can', 'each', 'itself', 'his', 'all', 'once', \n",
    "            'herself', 'more', 'our', 'they', 'hasn', 'on', 'ma', 'them', 'its', 'where', 'did', 'll', 'you', \n",
    "            'didn', 'nor', 'as', 'now', 'before', 'those', 'yours', 'from', 'who', 'was', 'm', 'been', 'will', \n",
    "            'into', 'same', 'how', 'some', 'of', 'out', 'with', 's', 'being', 't', 'mightn', 'she', 'again', 'be', \n",
    "            'by', 'shan', 'have', 'yourselves', 'needn', 'and', 'are', 'o', 'these', 'further', 'most', 'yourself', \n",
    "            'having', 'aren', 'here', 'he', 'were', 'but', 'this', 'myself', 'own', 'we', 'so', 'i', 'does', 'both', \n",
    "            'when', 'between', 'd', 'had', 'the', 'y', 'has', 'down', 'off', 'than', 'haven', 'whom', 'wouldn', \n",
    "            'should', 've', 'over', 'themselves', 'few', 'then', 'hadn', 'what', 'until', 'won', 'no', 'about', \n",
    "            'any', 'that', 'for', 'shouldn', 'don', 'do', 'there', 'doing', 'an', 'or', 'ain', 'hers', 'wasn', \n",
    "            'weren', 'above', 'a', 'at', 'your', 'theirs', 'below', 'other', 'not', 're', 'him', 'during', 'which']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "人工分词：\n",
    "\n",
    "这里，英文的分词，直接就是对着空白处分割就可以了。\n",
    "\n",
    "中文的分词稍微复杂点儿，具体可以百度：CoreNLP, HaNLP, 结巴分词，等等\n",
    "\n",
    "分词的意义在于，把我们的长长的字符串原文本，转化成有意义的小元素："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "texts = [[word for word in doc.lower().split() if word not in stoplist] for doc in doclist]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这时候，我们的texts就是我们需要的样子了："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['thursday',\n",
       " 'march',\n",
       " 'pm',\n",
       " 'latest',\n",
       " 'syria',\n",
       " 'aiding',\n",
       " 'qaddafi',\n",
       " 'sid',\n",
       " 'hrc',\n",
       " 'memo',\n",
       " 'syria',\n",
       " 'aiding',\n",
       " 'libya',\n",
       " 'docx',\n",
       " 'hrc',\n",
       " 'memo',\n",
       " 'syria',\n",
       " 'aiding',\n",
       " 'libya',\n",
       " 'docx',\n",
       " 'march',\n",
       " 'hillary']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "texts[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 建立语料库\n",
    "\n",
    "用词袋的方法，把每个单词用一个数字index指代，并把我们的原文本变成一条长长的数组："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dictionary = corpora.Dictionary(texts)\n",
    "corpus = [dictionary.doc2bow(text) for text in texts]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "给你们看一眼："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(51, 1), (505, 1), (506, 1), (507, 1), (508, 1)]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus[13]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这个列表告诉我们，第14（从0开始是第一）个邮件中，一共6个有意义的单词（经过我们的文本预处理，并去除了停止词后）\n",
    "\n",
    "其中，36号单词出现1次，505号单词出现1次，以此类推。。。\n",
    "\n",
    "接着，我们终于可以建立模型了："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lda = gensim.models.ldamodel.LdaModel(corpus=corpus, id2word=dictionary, num_topics=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以看到，第10号分类，其中最常出现的单词是："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.086*\"pm\" + 0.042*\"office\" + 0.033*\"secretarys\" + 0.024*\"meeting\" + 0.022*\"room\"'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda.print_topic(10, topn=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们把所有的主题打印出来看看"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  '0.009*\"week\" + 0.008*\"get\" + 0.008*\"next\" + 0.007*\"speech\" + 0.007*\"work\"'),\n",
       " (1,\n",
       "  '0.024*\"part\" + 0.022*\"release\" + 0.018*\"sent\" + 0.013*\"blackberry\" + 0.011*\"see\"'),\n",
       " (2,\n",
       "  '0.026*\"state\" + 0.018*\"bloomberg\" + 0.018*\"doc\" + 0.016*\"dialogue\" + 0.016*\"strategic\"'),\n",
       " (3,\n",
       "  '0.017*\"taliban\" + 0.008*\"woodward\" + 0.008*\"gender\" + 0.007*\"karzai\" + 0.005*\"amendment\"'),\n",
       " (4,\n",
       "  '0.012*\"assume\" + 0.010*\"im\" + 0.009*\"argentina\" + 0.009*\"strobe\" + 0.008*\"great\"'),\n",
       " (5, '0.008*\"would\" + 0.007*\"one\" + 0.007*\"said\" + 0.006*\"us\" + 0.005*\"new\"'),\n",
       " (6,\n",
       "  '0.020*\"haitian\" + 0.014*\"un\" + 0.013*\"prince\" + 0.008*\"know\" + 0.008*\"womens\"'),\n",
       " (7,\n",
       "  '0.020*\"yes\" + 0.009*\"know\" + 0.006*\"coming\" + 0.006*\"bill\" + 0.005*\"add\"'),\n",
       " (8,\n",
       "  '0.007*\"lauren\" + 0.007*\"madame\" + 0.007*\"get\" + 0.006*\"copy\" + 0.006*\"follow\"'),\n",
       " (9,\n",
       "  '0.011*\"mr\" + 0.010*\"ok\" + 0.009*\"mcchrystal\" + 0.009*\"would\" + 0.007*\"labour\"'),\n",
       " (10,\n",
       "  '0.086*\"pm\" + 0.042*\"office\" + 0.033*\"secretarys\" + 0.024*\"meeting\" + 0.022*\"room\"'),\n",
       " (11,\n",
       "  '0.031*\"israeli\" + 0.027*\"israel\" + 0.020*\"settlements\" + 0.015*\"palestinian\" + 0.012*\"settlement\"'),\n",
       " (12,\n",
       "  '0.008*\"us\" + 0.006*\"would\" + 0.005*\"new\" + 0.005*\"one\" + 0.005*\"people\"'),\n",
       " (13,\n",
       "  '0.012*\"january\" + 0.007*\"joanne\" + 0.006*\"good\" + 0.006*\"rolling\" + 0.006*\"laszczych\"'),\n",
       " (14,\n",
       "  '0.068*\"fyi\" + 0.022*\"cheryl\" + 0.021*\"pm\" + 0.021*\"fw\" + 0.017*\"mills\"'),\n",
       " (15,\n",
       "  '0.015*\"nuclear\" + 0.007*\"right\" + 0.007*\"iran\" + 0.005*\"would\" + 0.005*\"new\"'),\n",
       " (16,\n",
       "  '0.027*\"percent\" + 0.010*\"obama\" + 0.009*\"republicans\" + 0.008*\"unfavorable\" + 0.008*\"favorable\"'),\n",
       " (17,\n",
       "  '0.010*\"sounds\" + 0.009*\"good\" + 0.009*\"pm\" + 0.007*\"love\" + 0.006*\"would\"'),\n",
       " (18,\n",
       "  '0.009*\"said\" + 0.009*\"bill\" + 0.008*\"senate\" + 0.007*\"qddr\" + 0.007*\"health\"'),\n",
       " (19,\n",
       "  '0.036*\"call\" + 0.018*\"pls\" + 0.014*\"print\" + 0.012*\"tomorrow\" + 0.012*\"talk\"')]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda.print_topics(num_topics=20, num_words=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 接下来：\n",
    "\n",
    "通过\n",
    "```\n",
    "lda.get_document_topics(bow)\n",
    "```\n",
    "或者\n",
    "```\n",
    "lda.get_term_topics(word_id)\n",
    "```\n",
    "\n",
    "两个方法，我们可以把新鲜的文本/单词，分类成20个主题中的一个。\n",
    "\n",
    "*但是注意，我们这里的文本和单词，都必须得经过同样步骤的文本预处理+词袋化，也就是说，变成数字表示每个单词的形式。*\n",
    "\n",
    "### 作业：\n",
    "\n",
    "我这里有希拉里twitter上的几条(每一空行是单独的一条)：\n",
    "\n",
    "```\n",
    "To all the little girls watching...never doubt that you are valuable and powerful & deserving of every chance & opportunity in the world.\n",
    "\n",
    "I was greeted by this heartwarming display on the corner of my street today. Thank you to all of you who did this. Happy Thanksgiving. -H\n",
    "\n",
    "Hoping everyone has a safe & Happy Thanksgiving today, & quality time with family & friends. -H\n",
    "\n",
    "Scripture tells us: Let us not grow weary in doing good, for in due season, we shall reap, if we do not lose heart.\n",
    "\n",
    "Let us have faith in each other. Let us not grow weary. Let us not lose heart. For there are more seasons to come and...more work to do\n",
    "\n",
    "We have still have not shattered that highest and hardest glass ceiling. But some day, someone will\n",
    "\n",
    "To Barack and Michelle Obama, our country owes you an enormous debt of gratitude. We thank you for your graceful, determined leadership\n",
    "\n",
    "Our constitutional democracy demands our participation, not just every four years, but all the time\n",
    "\n",
    "You represent the best of America, and being your candidate has been one of the greatest honors of my life\n",
    "\n",
    "Last night I congratulated Donald Trump and offered to work with him on behalf of our country\n",
    "\n",
    "Already voted? That's great! Now help Hillary win by signing up to make calls now\n",
    "\n",
    "It's Election Day! Millions of Americans have cast their votes for Hillary—join them and confirm where you vote\n",
    "\n",
    "We don’t want to shrink the vision of this country. We want to keep expanding it\n",
    "\n",
    "We have a chance to elect a 45th president who will build on our progress, who will finish the job\n",
    "\n",
    "I love our country, and I believe in our people, and I will never, ever quit on you. No matter what\n",
    "\n",
    "```\n",
    "\n",
    "各位同学请使用训练好的LDA模型，判断每句话各自属于哪个potic\n",
    "\n",
    "么么哒"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
